In-Flight Mechanics: A Software Package Management Conversion Project

Over the course of most of 2009 I acted as the technical lead on a software package management conversion project in Yahoo Mail. I was part of a team of six people that ultimately reinstalled the software on most of the core servers and developed a completely new release management process. This paper documents the process we used, the problems we found, and the current state of the system as of mid-2010. The core of Yahoo Mail is approximately 7000 servers in datacenters around the world. These servers are grouped into farms of roughly 15 machines and associated back-end mail storage. Each farm stores the mail for a few hundred thousand to a few million users and acts as the central access point for a user’s Yahoo Mail experience. Traditionally these servers have been managed with a proprietary Yahoo dependency-based packaging system called yinst. Yinst operates similarly to RedHat rpm or Debian dpkg (although with more integration with a central software repository, so perhaps the best analogy is with yum or apt-get). In that system a release was just a list of packages to be installed on the hosts, with only minimal explicit considerations of dependencies. Any package dependencies not included in the initial install list would be automatically downloaded and installed as needed by yinst. The install process was based on an automated tool which would ssh to each host and launch yinst to request installation of the package list. This mechanism served Yahoo Mail well for a number of years. As the size of our installation grew, however, several deficiencies became apparent. Over time the task of managing our installed software and settings became more and more difficult. For example, the old system was completely additive – it was simple to create and install a new package, but extremely difficult to remove a package once it had been installed in production. This eventually led to more than 1300 packages installed on each server. Yinst includes per-package settings management as well as package management, but the existing system included no easy way to tie settings and packages together in a coherent release. The idea of rollback to a previous system state was also not well supported. To address these issues, a software tools group at Yahoo had been working for several years to develop a state-based package and setting management solution called Igor. Igor sits on top of yinst and enforces the exact package and setting list for a given host, through a versioned tagging mechanism. In essence this moves dependency checking from runtime to compile time (with compile time being the creation of the release). Igor had been in use for several years in other properties at Yahoo and appeared to work well. Thus in early 2009 my team was given the task of converting to Igor (igorizing) all the 7000 mail farm hosts. At the highest level this process included: